{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "15259778",
   "metadata": {},
   "source": [
    "# 格式转换\n",
    "在把数据从磁盘加载到系统内存的过程中，我们需要对数据进行打包和解包的操作，会有磁盘IO和网络IO等的开销。而在MindSpore中，我们可以把数据先归一化成特定的`MindSpore Record`的格式，其格式可以减少磁盘IO、网络IO的开销，从而获得更好的体验，这种格式还针对部分场景进行了性能优化。\n",
    "\n",
    "![conversion](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/data_conversion_concept.png)\n",
    "\n",
    "\n",
    "`MindSpore Record`数据格式具备的特征如下：\n",
    "\n",
    "1. 实现数据统一存储、访问，使得训练时数据读取更加简便。\n",
    "2. 数据聚合存储、高效读取，使得训练时数据方便管理和移动。\n",
    "3. 高效的数据编解码操作，使得用户可以对数据操作无感知。\n",
    "4. 可以灵活控制数据切分的分区大小，实现分布式数据处理。\n",
    "\n",
    "如下图所示，`MindSpore Record文件`由`数据文件`和`索引文件`组成。\n",
    "\n",
    "![MindSpore Record](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/mindrecord.png)\n",
    "\n",
    "其中数据文件包含文件头、标量数据页、块数据页，用于存储用户归一化后的训练数据。\n",
    "\n",
    "而索引文件则包含基于标量数据（如图像Label、图像文件名等）生成的索引信息，用于方便的检索、统计数据集信息。\n",
    "\n",
    "数据文件中的文件头、标量数据页、块数据页的具体用途如下所示：\n",
    "\n",
    "- `文件头`：是MindSpore Record文件的元信息，主要用来存储文件头大小、标量数据页大小、块数据页大小、Schema信息、索引字段、统计信息、文件分区信息、标量数据与块数据对应关系等。\n",
    "- `标量数据页`：主要用来存储整型、字符串、浮点型数据，如图像的Label、图像的文件名、图像的长宽等信息，即适合用标量来存储的信息会保存在这里。\n",
    "- `块数据页`：主要用来存储二进制串、NumPy数组等数据，如二进制图像文件本身、文本转换成的字典等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c3233a",
   "metadata": {},
   "source": [
    "## 一、转换成MindSpore Record格式\n",
    "使用`mindspore.mindrecord`接口可以将数据转换成record格式。此接口提供了一些方法将不同数据集转换为MindRecord格式， 也提供了一些操作MindRecord数据文件的方法如读取、写入、检索等。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d8f50aa4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MSRStatus.SUCCESS"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 此处设计一些进阶计算机知识，重点关注如何将数据转换成MindSpore Record格式\n",
    "import os\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "import mindspore.mindrecord as record\n",
    "\n",
    "# 定义文件路径\n",
    "dir = \"./test.mindrecord\"\n",
    "\n",
    "if os.path.exists(dir):\n",
    "    os.remove(dir)\n",
    "    os.remove(dir + \".db\")\n",
    "\n",
    "# 定义包含的字段\n",
    "schema = {\"file_name\": {\"type\": \"string\"},\n",
    "          \"label\": {\"type\": \"int32\"},\n",
    "          \"data\": {\"type\": \"bytes\"}}\n",
    "\n",
    "# 声明MindSpore Record文件格式\n",
    "writer = record.FileWriter(file_name=dir)\n",
    "writer.add_schema(schema)\n",
    "writer.add_index([\"file_name\", \"label\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9670b0",
   "metadata": {},
   "source": [
    "- `FileWriter`操作可以将用户自定义的数据转为MindRecord格式数据集，参数`file_name`为生成的MindRecord文件路径。\n",
    "- `add_schema`操作可以增加描述用户自定义数据的schema，参数为schema内容的字典格式。\n",
    "- `add_index`操作可以指定schema中的字段作为索引来加速MindRecord文件的读取。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "65f0fb5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建数据集\n",
    "# 创建数据集\n",
    "data = []\n",
    "for i in range(100):\n",
    "    i += 1\n",
    "    sample = {}\n",
    "    white_io = BytesIO()\n",
    "    Image.new('RGB', (i*10, i*10), (255, 255, 255)).save(white_io, 'JPEG')\n",
    "    image_bytes = white_io.getvalue()\n",
    "    sample['file_name'] = str(i) + \".jpg\"\n",
    "    sample['label'] = i\n",
    "    sample['data'] = white_io.getvalue()\n",
    "\n",
    "    data.append(sample)\n",
    "    if i % 10 == 0:\n",
    "        writer.write_raw_data(data)\n",
    "        data = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a58d41de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MSRStatus.SUCCESS"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 转换数据\n",
    "if data:\n",
    "    writer.write_raw_data(data)\n",
    "writer.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ae66f4d",
   "metadata": {},
   "source": [
    "- `commit`操作可以将内存中的数据同步到磁盘，并生成相应的数据库文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29b24387",
   "metadata": {},
   "source": [
    "## 二、读取MindRecord格式文件\n",
    "通过MindDataset接口可以读取MindSpore Record格式文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "586dd7e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 100 samples\n"
     ]
    }
   ],
   "source": [
    "import mindspore.dataset as ds\n",
    "import mindspore.dataset.vision as vision\n",
    "\n",
    "# 读取文件\n",
    "dataset = ds.MindDataset(dataset_files=dir)\n",
    "# 对图片文件解码\n",
    "decode = vision.Decode()\n",
    "dataset = dataset.map(operations=decode, input_columns=['data'])\n",
    "\n",
    "# 样本计数\n",
    "print(\"Got {} samples\".format(dataset.get_dataset_size()))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mindspore",
   "language": "python",
   "name": "mindvision"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
